Papers with video-and-language tasks
Enhancing Temporal Modeling of Video LLMs via Time Gating (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing Video Large Language Models neglect temporal information in video data, leading to struggles with temporal-aware video understanding. |
| Approach: | They propose a Time Gating Video LLM (TG-Vid) that employs a time gating module to enhance temporal modeling. |
| Outcome: | The proposed model outperforms existing Large Language Models on video-and-language tasks and ablation studies show that the model outpersforms the existing models. |
DeCEMBERT: Learning from Noisy Instructional Videos via Dense Captions and Entropy Minimization (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to train models on unlabeled web videos are noisy and temporally misaligned . authors propose a method that adds captions and constrained attention loss to improve performance . |
| Approach: | They propose a method that adds captions from video frames as auxiliary text input to provide visual cues for learning better video and language associations. |
| Outcome: | The proposed method outperforms state-of-the-art methods on video-and-language tasks . it adds captions and constrained attention loss to improve model performance . |